Use of Generative Artificial Intelligence for Development of Plain Language Summaries: A Blinded Assessment of Education Preferences of the Sickle Cell Disease Community

Introduction: Sickle cell disease (SCD) is an inherited condition that reduces life expectancy and has a profound impact on quality of life. Assessment of social media conversations in the SCD community in the UK highlighted health inequities, including issues with access to emergency care, low levels of healthcare professional empathy, and racial bias/stigmatization (Shastri O, et al. ASH 2023; abstract 1057). This emphasizes the urgent unmet need for additional education. Use of generative artificial intelligence (GenAI) to facilitate the development of medical content, including plain language summaries (PLS) of research may increase efficiency, reduce resource cost and ultimately improve accessibility to educational information across a range of audiences. We assessed the ability of GenAI to develop a PLS of our social media listening study.

Methods: We developed 3 written versions of a PLS of this study: human-written by a medical writer, AI-generated, and hybrid AI-human, where a person living with SCD edited the AI version for readability. Each was ~300-400 words and with a target reading age of 12 years. The AI PLS was developed using Pfizer's GenAI tool, MAIA (Medical AI assistant). A video version of each written PLS was developed using the AI tool, Synthesia. People with SCD and their carers (≥18 years of age) were recruited via telephone to complete an online survey to assess the understandability of the 3 written PLS and preference for written versus video PLS. Participants were presented with 1 of the 3 written PLS at random and asked to assess how easily they understood it on a 5-point scale (1=very difficult; 2=difficult; 3=neither difficult nor easy; 4=easy; 5=very easy). They were then asked 3 multiple-choice questions to gauge their understanding. Participants then rated the other 2 written PLS and were asked to rank all 3 in order of most easily understood. Finally, participants watched the video version of their top-rated written PLS and stated which format they preferred. Participants were blinded to PLS source. The Flesch-Kincaid calculator (https://goodcalculators.com/flesch-kincaid-calculator/) was used to provide an objective measure of readability for each PLS.

Results: Of 93 participants, there were 88 living with SCD and 5 caring for someone with SCD. The AI versions of the PLS achieved similar scores for understandability to the human-written version: mean ± standard deviation understandability scores were 4.1 ± 0.9 (human), 4.0 ± 0.9 (AI), and 3.9 ± 0.8 (AI-hybrid). Overall, 81% of participants identified the human PLS as easy or very easy to read, similar to 76% for the AI PLS, and 74% for the AI-hybrid PLS. Overall, 41 participants (44%) ranked the human PLS in first place for understandability, 31 (33%) the AI PLS, and 21 (23%) the AI-hybrid PLS. For the multiple-choice questions, results were similar regardless of which PLS participants saw first, with over 85% correctly identifying the main findings of the study and the conclusions of the author; however, 63% incorrectly thought the data on which the PLS was based were obtained from interviewing people affected by SCD rather than social media listening. Fifty-four participants (58%) preferred the video PLS over the written PLS, 27 participants (29%) preferred the written PLS and 12 (13%) had no preference. Flesch-Kincaid scores for the three PLS were as follows: human (reading ease score, 62; reading level, 8^th to 9^th grade); AI (reading ease score, 58; reading level, 10^th to 12^th grade); AI-hybrid (reading ease score, 63; reading level, 8^th to 9^th grade).

Conclusion: There is a clear need for additional resources and education in SCD, which may be supported by the development of PLS. The limited studies that have assessed the capabilities of AI to generate PLS to date have focussed on clinical research. We have now expanded this to assess use of AI to develop PLS from a social media listening study that evaluated real-world experiences of the UK SCD community. Our study suggests that GenAI can generate PLS that are as informative as conventional, human-written PLS, and achieve similar readability scores as judged by people living with SCD in the UK (mean 4.1 for human-written, 4.0 for AI and 3.9 for AI-hybrid). We propose that GenAI may offer an alternative to conventional human-written PLS, providing a time- and resource-efficient solution to increase accessibility to educational resources.

Shastri:Pfizer Ltd: Current Employment. Agrippa:Pfizer Ltd: Consultancy; Sanius: Current Employment. Moss:Prime: Current Employment. Millard:Pfizer Ltd: Ended employment in the past 24 months. More:Pfizer Ltd: Current Employment. Brown:Pfizer Ltd: Current Employment. Sharif:Pfizer Ltd: Current Employment. Finch:Prime: Current Employment.